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I . INTRODUCTION 



A. PURPOSE OF STUDY 

The purpose of this study was to assess the character- 
istics of the U.S. Navy officer students registered in the 
Operations Research/System Analysis curriculum at the U.S. 
Naval Postgraduate School by means of a Biographical Ques- 
tionnaire, the Graduate Record Examination and the Strong 
Vocational Interest Blank and determine if these measures 
offered a means of improving the selection of U.S. Naval 
officers for enrollment in the Operations Research/Systems 
Analysis curriculum. 

This study was a portion of the NPS Student Selection 
Project sponsored by' the Navy Personnel Research and Develop- 
ment Center, concerning the prediction of officer/student 
academic performance and satisfaction in graduate curricula 
at the U.S. Naval Postgraduate School. 

Only academic performance was investigated in the study 
since two previous studies, conducted by Cook [1] and Sofge 
[2], found that none of the predictors of satisfaction 
cross-validated at a statistically significant level. 

B. U.S. POSTGRADUATE SELECTION PROCESS 

The Postgraduate Education Selection Board in the Navy's 
Bureau of Personnel (BuPers) is in charge of selecting Navy 
Officers best suited for courses in the postgraduate program 
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based on the individual's preference, for postgraduate curri- 
cula, professional performance, prior academic performance, 
promotion potential and the needs of the service [3] . 

The needs of the service are determined by the Chief of 
Naval Personnel and sent to the Board annually. 

An officer's prior academic performance is translated 
into an Educational Potential Code (EPC) based upon an evalua- 
tion of previous grades. The EPC is supposed to be weighted 
40% by the Board. 

An officer's professional performance is supposed to be 
weighted 60%. The Officer's Fitness Reports are used as 
the primary indicator of professional performance [3] . 

Upon completion of the selection process, the Board sends 
a list of principle and alternate selectees to the Officer 
Detailing Section of BuPers. The Detailers then decide which 
of these selected officers are available for education. A 
more detailed description of this selection procedure is 
provided by R.S. Elster [3], 

C. DEFINITION OF KEY TERMS 

1. The Quality Point Rating (QPR) , is a student's 
weighted grade score computed from standards established by 
the Naval Postgraduate School [4] as follows: 
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Grade 



Point Value 



A 




4 .0 


A- 




3.7 


B+ 




3.3 


B - 




3.0 


B- 




2.7 


C+ 




2.3 


C 




2.0 


c- 




1.7 


D+ 




1.3 


D 




1.0 


X 




0.0 



Multiplying the credit-hours value of a course by the 
point value received provides the total quality points for 
that course. Adding the quality points for all courses and 
dividing by the total number of credit hours results in a 
figure defined as a Quality Point Rating (QPR) . For example, 
if a stuaent receiveo four hours of A, four hou] 
four hours of B, his QPR would be: 



4 X 4 + 4 X 3.7 +4X3 - c _ 

— 0 . 0 / 

4 + 4 + 4 

2. The Graduate Record Examination is a widely accepted 

paper and pencil aptitude test designed to predict academic 
potential at the graduate level. The GRE is a secure test 
which is administered under controlled conditions and yields 
two scores: verbal ability (GREV) and quantitative ability 

(GREQ) [5]. It is prepared and published by the Education 
Testing Service of Princeton, New Jersey. 

3. The Strong Vocational Interest Blank (SVIB) , is a 
test to assess individual interests as related to interests 
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of incumbants in various occupations [6] . The SVIB contains 
399 items, listing school subjects, hobbies and activities 
to which the individual's response is either like, dislike, 
or indifferent. 

4. Biographical Questionnaire. This questionnaire was 
designed to obtain historical/biographical facts from officers/ 
students as well as attitudes toward postgraduate education 
and its place in today's U.S. Navy. The questionnaire con- 
tained 61 questions which required "yes" or "no" responses, 

and was developed by R.A. Weitzman with the assistance of 
J.L. Cook [1] . 

5. Correlation Coefficient. The Simple Correlation 
Coefficient is a measure of linear association between two 
random variables Y and X [7] . The Multiple Correlation 
Coefficient is a measure of linear association between a 
random variable Y and a set of random variables 

X = (X 1 ,X 2 ,. . . ,X ) [7] . 

The range of possible values of the Correlation Coeffi- 
cient is from -1.00 to +1.00. A correlation of 0.00 indi- 
cates that knowledge of the value of X gives no information 
about Y. 

6. The Reliability Coefficient, is a measure of consis- 
tency or stability of test results. It is often obtained- 
by inter-correlating the test with the results if the test 
is retaken in the same or alternate form, or it is obtained 
by splitting the test into two parts and intercorrelating the 
two sets of scores [8], 
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7. Validity is the effectiveness of the test in repre- 
senting, describing or predicting the attribute that the 
test purports to measure [8]. 

8. Cross-Validation: Testing out of a set of items or 

system of test weights derived from one sample, to see to 
what extent the procedure or equation retains its validity 
with a new and independent sample. Cross-Validation is 
especially important when items or test weights have been 
chosen from a large number of possible alternatives, and when 
the original sample was small [8], 

D. ASSUMPTIONS 

It was assumed that academic performance, represented by 
Quality Point Rating is related to some predictors, such as 
biographical items, prior academic performance, SVIB scores 
or GRE scores. 

E. LIMITATION 

This study was limited to male, U.S. Navy Officer Students 
who were enrolled in the Operations Research/Systems Analysis 
curriculum in Spring 1974. 
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II. DESCRIPTION OF INSTRUMENTS 



A. GRADUATE RECORD EXAMINATION (GRE) 

This test is used in efforts to determine student's apti- 
tudes for postgraduate study. Included in the test are ver- 
bal reasoning questions, reading comprehensive questions and 
various kinds of mathematical problems involving arithmetic 
reasoning, algebra and the interpretation of graphs, diagrams 
and descriptitve data [5], Established reliabilities for 
the test are: [9] 



The GRE has increased in utilization as a selection tool. 
There were 22,000 candidates tested in 1958 and the number 
has increased since then. Over 300,000 candidates were 
tested in 1973 [10]. 

Previous studies of the validity of the GRE in predicting 
academic performance at the U.S. Naval Postgraduate School 
were conducted by Kauder and Ebert in 1963, Dreese and Russel 
in 1964, and Cook in 1974. The first two studies obtained 
over 0.45 validation correlations while the later one obtained 
a correlation of 0.37 with a relatively small sample size 
(N = 30) . 



Test 



Reliability Coefficient 



Verbal 

Quantitative 



0.90 

0.89 
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B. STRONG VOCATIONAL INTEREST BLANK (SVIB) 

The primary use of the Strong Vocational Interest Blank 
is for counseling high school and college students about 
their career choices. However this test can also be used 
for personnel selection. When used by a professionally 
trained person, the SVIB can provide useful information for 
evaluating an applicant [6] . 

The basic SVIB documents are a test booklet containing 
the items, an answer sheet where an individual records his 
responses, and the profile forms that are used to report 
the results. 

The SVIB has two types of scales, occupational scales and 
non-occupational scales. People in different occupations 
have different interests. The SVIB is a device to identify 
such differences among those occupations. The SVIB scores 
are used to counsel individuals regarding occupations in 
which they would be satisfied. The SVIB accomplishes this 
by providing an index of the similarity between a person's 
interests and those of men (or women) in each of a wide 
range of occupations. Basically, the technique used in SVIB 
is to present the individual with a long list of activities 
and ask him to indicate which he likes and dislikes. The 
answers are analyzed by comparing his responses to those of 
men (or women) already established in a wide range of activi- 
ties. If his choices coincide with say, engineers, then he 
received a high score on the engineers scale. The mean score 
for a particular occupation has been set at 50 with a standard 
deviation of 10. 



13 



The Academic Achievement scale (AACH) as one of the non- 
occupational scales is an attempt to identify the pattern of 
interests associated with good scholarship. The scale inclu- 
ded items that differentiate between good and poor students. 
It has been shown to be moderately effective in predicting 
grades [6 ] . 

C. BIOGRAPHICAL QUESTIONNAIRE 

The questionnaire was developed to obtain the following 
information: 

1. Source of commission and/or prior enlisted 
service . 

2 . Rank . 

3. Undergraduate institution attended, degree 
received, undergraduate major and grade point 
average . 

4. Warfare specialty. 

5 . Race . 

6. Height and weight. 

7. Religion. 

8. Marital status and/or sex dependents. 

9. Birth-order of subject. 

10. Educational level of father. 

11. Military career of father. 

12. High school background information. 

13. Personal habits. 

14. Participation in Boy Scout and rank attained. 

15. Designation by Postgraduate Selection Board. 
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16. Satisfaction with the Naval Postgraduate 
School and/or curriculum. 

17. Possible future use of skills obtained at the 
Naval Postgraduate School. 

18. Mathematics background. 

19. Motivation for coming to the Naval Postgraduate 
School . 

The questionnaire utilized a booklet with 61 questions, 
each of which required "yes" or "no" responses. The responses 
were recorded on a separate sheet. The answer sheet also 
contained an administrative section which required information 
such as: Social Security number (for identification) , 

current curriculum, number of quarters completed in current 
curriculum, and QPR. 
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III. METHODS , TECHNIQUES AND PROCEDURES 



A. GENERAL 

Seventy-two U.S. Navy officer students in the Operations 
Research/Systems Analysis curriculum at NPS who were at 
various stages in their curriculum when the data were collected 
(quarter 3, Academic Year 73-74) constituted the population 
of this study. A random sample of 50 students was randomly 
selected from the population for the development of predic- 
tors of academic performance. The remaining sample constituted 
the cross-validation sample. 

B. LINEAR REGRESSION ANALYSIS 

The purposes of Linear Regression are to select the set 
of explanatory variables which should be included in a model, 
and then to estimate the regression coefficients which 
describe the quantitative weights of these variables. 

The model to be used, in matrix notation, has the form: 

Y = XB + e 

where 



Y,1 
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It was assumed that 

1. E(e) = 0, hence E ( Y) = XB 

2. The components of e are identically and indepen- 
dently distributed, hence the variance covariance 
matrix for the vector variables e is 



E (e e ' ) 



C 2 I 

nxn 



The Gauss-Markov theorem states that "the least squares 
estimators have minimum sampling variance in the class of 
all unbiased estimators which are linear functions of Y . " 

To obtain the least square estimators, one minimizes e'e, 
where e = Y - XB, with respect to B. This yields: 

A -1 
B = (X'X) X'Y , 

A 

where B is the estimator of B, provided that (X'X) is 
non-singular . 

To test the hypothesis of B = B* against the alternative 
hypothesis B ^ B* , one uses Analysis of Variance criteria. 
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All Analysis of Variance tests are based on what is known 
as the Likelihood Ratio test. 

In the case of the model 

Y = XB + e , 

we need another assumption, that is e is distributed as 

2 

multivariate normal with mean 0^ and variance C I, hence 

Y ~ N (XB , C 2 I) 

and the likelihood function has the form: 

— — i(Y - XB) ' (Y - XB) 

2 1 2 r 

L(Y,B,0 = — - -jj e 

(2tt) n/ ^ 

Consider maximizing the likelihood function with respect 

2 . . 
to B and C under two situations. The first case is when 

2 

we allow B and C to take on any value they wish m the range 
2 

(-°° <_ B _< =°; C >_ 0) . Denote this range ft. This is the 
unrestricted case and the maximum value of the likelihood 

A 

function will be denoted L(ft) . The second case is when the 

2 

restriction on the parameters B = B * ; C ^.0. This range is 
denoted by W and the maximum value of the likelihood function 

A 

under the restriction of the hypothesis is denoted L(W) . We 
would reject the hypothesis B = B* in favor of alternative ' 
hypotheses B = B* if in fact: 



L 



= MW) < 
L (ft) 



k 
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for some constant k < 1 where k is chosen such that the 
probability of rejecting the hypothesis when it is true is 
equal to a (usually a = 5 or 10%) . That is, select k such 
that 



a = / g(LjHo is true) dL 

— 00 

where g(L|Ho true) is the distribution of the likelihood 
ratio when Ho is true. 

2 2 
Maximizing L(Y,B,C ) with respect to B and C under ft 

and W gives the likelihood ratio 

(Y - XB) ' (Y - XB) n/2 

L = [ ] 

(Y - XB*) ' (Y - XB*) 

With B = (X , X)" 1 X'Y. 

The numerator of the likelihood ratio (ignoring the 
power n/2) may be expressed as 

(Y - XB*) (I - XS^X') (Y - XB*) , 



where 



S" 1 = (X'X) -1 . 
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The denominator of the likelihood ratio may be expressed 



as : 



(Y - XB*) ' (I - XS~ 1 X') (Y - XB*) + (Y - XB*) ' (XS“ 1 X) (Y - XB*) 



Hence, L may be written with Z = (Y - XB*) as: 

Z'A Z n/2 

L = [ ] 

Z'A Z + Z'A 2 Z 



where 



A 1 = (I - XS -1 X') 

a 2 = (xs - 1 x') 

and it also can be proven that A^ and are idempotent 
2 2 

matrices (A. = A^, A 2 = A 2 ) with rank (n - p) and p 

respectively. 

Since Y ~ N(XB,C 2 I) then Z - N (XB - XB * , C 2 I) and also 
since A^ and A 2 are idempotent and A^A 2 = 0 then 

Z'A^ Z'A 2 ZL 

— 5 and — « 

C C 

are independent Chi-Squares: 

Z'A Z 9 

— ~ X (n - P) 

C 
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and 



Z'A~Z „ 

— 2 — " X IP?*) / 

C 

where X is the non centrality parameter and its value is 
equal to 

X = (B - B*) 1 X ' X (B - B*) . 

Note, that although the ratio 

Z , A 1 Z/C 2 

Z'A,Z/C 2 + Z'A n Z/C 2 

JL Z 

is not the ratio of two independent Chi-Squares and hence 
not distributed F, the statistic 

(Z'A Z/C 2 )/p 

U = ^ 

(Z'A 1 Z/C )/n-p 

is the ratio of two independent Chi-Squares divided by their 
respective degrees of freedom. Moreover, the denominator is 
distributed as central Chi-Square and the numerator is dis- 
tributed as non central Chi-Square and, hence, the ratio is 

distributed as F(p,n-p,X). If the hypothesis B = B* is true, 

1 X 1 X 

the noncentrality parameter X = -^-(B - B*) — *-(B - B*) is zero 

Z C 

and the U statistic is distributed as central F(p,n-p). 
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To show that the F test and the likelihood ratio test 



are equivalent, we note that 



L 



± ] 

Z'A 1 Z + Z'A 2 Z 



Z'A^ n/2 




n/2 



U and L are monotonically related in a manner given in the 
graph below. 



The likelihood ratio test is said to reject Ho when L < k 
where k is chosen to give a significant test of level a. But 
this is identical to rejecting Ho when U > k' where k' is 
chosen to give a significant test of the same level and 
since we know the distribution of U when Ho is true we can 
select k' from the appropriate F table [11]. 

In many situations, there is not sufficient information 
about the order of importance of the independent variables 



L 




k ' 



U 
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X- X in predicting the dependent variable Y. Testing 

1 .P 

Ho: B^ = 0 for each variable X^ does not reveal this ordering. 

For instance, rejecting the Ho that = 0 while accepting 
the remaining B^ = 0 could have led to the false conclusion 
that X^ was the only variable of importance in predicting Y 

[7]. 



One solution is the Stepwise Regression procedure which 
selects a best subset of predictors according to the following 
procedures: The first step selects the single variable which 

best predicts Y. The second step finds the variable which 
best predicts Y, given the first variable entered. In the 
steps that follow, either a variable is entered which best 
improves the prediction of Y given all the variables entered 
from the previous step, or a variable is removed from the 
set of predictors if its predictive ability falls below a 
given level [7] . The process is terminated when no further 
variable improves the prediction of Y. 



C. JACKKNIFING 

Least square estimations provide unbiased estimators as 
long as the specified model is correct. But, at the early 
stages of regression the model is not complete. Hence the 
estimators may be biased. Generally this bias diminishes 
as the sample size increases. One way to eliminate this is 
to use the Jackknife procedure. 

The rationale behind the Jackknife is as follows [12] : 

A 

Let G, a biased estimate of a parameter 0, decrease linearly 
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with sample size (a common situation.) . Suppose 

/s g A 

E ( 0 ) = 0 + r=— where Nm = sample size. Let 0 be an estimate 
Nm o 

A A A 

of 0 based on all Nm observations. Let 6-^ / 0 2 / • • • r be a 
corresponding estimate based on (N-l)m observations, and 



e (e ± ) 



e + (N-l)m * 



Let 



J i (0) = N0 - (N-l) 0 ± . 



Then 



E(J i (0) ) 



NE(0) - (N-l)E(0 i ) 



= N(e+ tl; ) - (N - 1><e+ 



= N0 + | - (N-l) - - 

m m 



= 0 



J(0) = i Z J. (0) 
i=l 1 



is the best unbiased estimate of 0 with variance equal to 



V ( J ( 0 ) ) = ^ V(J. (0) ) 
N l 



and 
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asymptotically 



J(0) 



N(0,V(J(0)) . 



For testing Ho: 0 = 0 vs Ha: 0^0, use the t statistic 



The Jackknife procedure has the advantages of being usable 
on a relatively small data base and of theoretically adjusting 
for bias. 

In applying this method to multiple regression, divide 
the sample data into N subsamples of equal size m. Based 
on the Stepwise Regression, investigate the first variable 
to enter into the equation and obtain N estimates of the 

A A 

parameters and the corresponding pseudo-value J^(0) and J(0) 
and use the t statistic to test the hypothesis regression 
coefficient is equal to zero against the alternative hypothe- 
sis that the regression coefficient is not equal to zero. 

If we reject the Ho, the next step would be to investigate 
the second variable . Repeat the same procedure until we 
reject the Ho of including a variable into the equation or 
alternately all variables to be considered are accepted at 
a statistically significant level to be included into the 
equation . 

Final estimates of the parameters are obtained from 
the next to final step or final step if all variables 
considered are included in the equation. 



t 



0 



J(6) 
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D. PATTERN ANALYSIS 

Identifying responses to items as correct or incorrect 
makes it possible to determine test scores by counting the 
number of items answered correctly. However, the total num- 
ber of items answered correctly does not exhaust all the 
information contained in the responses. Different indivi- 
duals may answer different combinations of items correctly 
although the total number of items answered correctly is the 
same. To each of these patterns, there corresponds a differ- 
ent possible score, called a pattern score. On a three-item 
test, for example, there are 8 different possible pattern 
scores. In contrast, there are only 4 different scores 
determinable by counting the number of items answered correct 
ly. By creating a more refined and extensive measurement 
scale, pattern scores may produce higher validities than 
traditional total-correct scores [13] . 

Pattern scores are also possible for different patterns 
of any binary responses, including yes or no responses to 
items on a biographical inventory. 

A computer program was developed by B.F. Folce Jr. [14] 
for his Master's thesis under the guidance of R.A. Weitzman. 
It is called Pattern Analysis Item Nominator (PAIN) . PAIN 
operated by computing a mean criteria score for each pattern 
of responses in a given subset, assigning these scores to 
subjects having that pattern of responses, and correlating 
the assigned scores with the subject's actual criterion score 
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IV. DATA PRESENTATION 



A. GENERAL 

The primary analysis of the data was conducted using 
three packaged computer programs at the W.R. Church Computer 
Center at the Naval Postgraduate School. These programs were 
the Statistical Package for the Social Sciences (SPSS) to 
provide Stepwise Multiple Regression as well as Pearson 
Correlations, BIMED03R to provide Multiple Regressions for 
the data within each selection of subsamples from the same 
population, and SNAP/IEDA to obtain scatter-plots of the data. 

Two programs written by B.F. Folce were used in Pattern 
Analysis, The first program called Pattern Analysis Item 
Nominator (PAIN) provides the following information: 

1. Validities of all item subsets examined. 

2. A list of the items that form the most valid 
subset of a given size. 

3. The validities of the most valid subset of each 
size . 

The second program of Folce' s was used to cross-validate 
the items selected by PAIN. 

B. CRITERIA AND PREDICTORS 

1 . Criteria 

The performance criterion used in this study was 
standardized QPR, denoted by ZQPR. This standardization was 
made to ensure that the differences among students would be 
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